Multi-field CCD capture for HDR imaging

ABSTRACT

Techniques are described to combine image data from multiple images with different exposures into a relatively high dynamic range image. A first image of a scene may be generated with a first operational mode of an image processing system. A second image of the scene may be generated with a second different operational mode of the image processing system. The first image may be of a first spatial resolution, while the second image may be of a second spatial resolution. For example, the first spatial resolution may be higher than the second spatial resolution. The first image and the second image may be combined into an output image of the scene. The output image may be of a higher dynamic range than either of the first image and the second image and may be of a spatial resolution higher than the second spatial resolution.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.13/440,902, filed on Apr. 5, 2012, which claims the benefit of priorityto U.S. Provisional Patent Application Ser. No. 61/472,495, filed onApr. 6, 2011, each of which is hereby incorporated by reference in itsentirety for all purposes.

TECHNOLOGY

The present invention relates generally to image generation, and inparticular, to generating high dynamic range images.

BACKGROUND

Real world scenes may have contrast ratios of as much as 50,000:1between the brightest highlights and the darkest shadows. Many existingimage processing systems are only capable of reproducing contrast ratiosof at most a few hundreds to one. Thus, many existing cameras are notcapable of taking advantage of the fact that display systems may now becapable of supporting high dynamic range (“HDR”) image formats andpermitting rendering images of contrast ratios of a few thousands to oneor better.

HDR is not yet widely supported because of the high costs associatedwith HDR image acquisition under existing approaches. For example, imagesensors that are capable of producing discerning responses to a widerange of luminance levels are expensive to manufacture. Even if deployedat a high cost, a large volume of image data that would be generated,requiring expensive processing power to process the image dataresponsively. As a result, for small footprint devices such as consumerelectronic devices, it would be neither economically feasible nortechnically viable to support HDR image acquisition.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection. Similarly, issues identified with respect to one or moreapproaches should not assume to have been recognized in any prior art onthe basis of this section, unless otherwise indicated.

BRIEF DESCRIPTION OF DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1A illustrates an example image processing system, in accordancewith some possible embodiments of the present invention;

FIG. 1B illustrates an example light sensitive area of an image sensor,in accordance with some possible embodiments of the invention;

FIG. 2A and FIG. 2B illustrate example operations of an image processingsystem for capturing multiple images of a scene, in accordance with somepossible embodiments of the present invention;

FIG. 2C and FIG. 2D illustrate example timelines for capturing multipleimages of a scene, in accordance with some possible embodiments of thepresent invention;

FIG. 3A illustrates example operations of an image processing system forcapturing multiple images of a scene, in accordance with some possibleembodiments of the present invention;

FIG. 3B illustrates an example light sensitive area that may be used byan image processing system to support capturing multiple images of ascene in different operational modes, in accordance with some possibleembodiments of the invention;

FIG. 4A, FIG. 4B and FIG. 4C illustrate example process flows, accordingto a possible embodiment of the present invention; and

FIG. 5 illustrates an example hardware platform on which a computer or acomputing device as described herein may be implemented, according apossible embodiment of the present invention.

FIG. 6 illustrates an example histogram for predicting missing colors,according a possible embodiment of the present invention.

DESCRIPTION OF EXAMPLE POSSIBLE EMBODIMENTS

Example possible embodiments, which relate to HDR image processingtechniques, are described herein. In the following description, for thepurposes of explanation, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, that the present invention may be practicedwithout these specific details. In other instances, well-knownstructures and devices are not described in exhaustive detail, in orderto avoid unnecessarily occluding, obscuring, or obfuscating the presentinvention.

Example embodiments are described herein according to the followingoutline:

-   -   1. GENERAL OVERVIEW    -   2. IMAGE PROCESSING SYSTEM    -   3. LIGHT SENSITIVE AREA    -   4. OPERATIONS FOR CAPTURING MULTIPLE IMAGES    -   5. MIXED OPERATIONAL MODES FOR CAPTURING MULTIPLE IMAGES    -   6. MIXED OPERATIONAL MODES IN LIGHT SENSITIVE AREA    -   7. ADDITIONAL EXPOSURE TO COMPLEMENT MISSING COMPONENT COLOR    -   8. PREDICTING MISSING COMPONENT COLOR    -   9. EXAMPLE PROCESS FLOW    -   10. IMPLEMENTATION MECHANISMS—HARDWARE OVERVIEW    -   11. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

1. GENERAL OVERVIEW

This overview presents a basic description of some aspects of a possibleembodiment of the present invention. It should be noted that thisoverview is not an extensive or exhaustive summary of aspects of thepossible embodiment. Moreover, it should be noted that this overview isnot intended to be understood as identifying any particularlysignificant aspects or elements of the possible embodiment, nor asdelineating any scope of the possible embodiment in particular, nor theinvention in general. This overview merely presents some concepts thatrelate to the example possible embodiment in a condensed and simplifiedformat, and should be understood as merely a conceptual prelude to amore detailed description of example possible embodiments that followsbelow.

Under techniques as described herein, multiple images with differentexposures may be captured back to back to create an HDR image. In somepossible embodiments, an image sensor such as a charge-couple-device(CCD) type may be relatively slow to offload a full resolution image ina reasonable amount of time. As a result, it may be relatively difficultto merge multiple images of the same scene with such an image sensor. Insome possible embodiments, techniques herein may be used to reduce thetime of capturing multiple images with different exposures with theimage sensor.

In some possible embodiments, one of the multiple images of the samescene may be set to, for example, an optimal exposure with a fullresolution. This image may be used to capture either shadow area details(for example, with a relatively longer exposure) or bright area details(for example, with a relatively short exposure). In some possibleembodiments, another of the multiple images may be set to a lowerspatial resolution. This other image may be set to capture luminancelevels expressed in bright areas if the image with the optimal exposureis set to capture the shadow area details. On the other hand, this otherimage may be set to capture luminance levels expressed in shadow areasif the image with the optimal exposure is set to capture the bright areadetails. In various possible embodiments, additional images may or maynot be captured of the same scene for the purpose of combining into anHDR image.

In some possible embodiments, an alternate video mode such as a previewmode, a view finder mode, etc. may be used as the lower resolution modementioned above to capture low resolution images. Techniques herein maysupport switching between multiple operating modes from the fullresolution to the lowest resolution modes. In some possible embodiments,a lower resolution image may be upsampled into a stack frame comprisingthe same number of pixels as the higher resolution image. In somepossible embodiments, the upsampled lower resolution image and thehigher resolution image may be merged to produce the HDR imagecomprising additional dynamic range luminance information.

In various possible embodiments, single-field, two-field, three-field,four-field, etc., image sensors may be used in an image processingsystem herein. One or more electronic shutter operations and/ormechanical shutter operations that transfer/shift measurable imageryresponses from individual fields of the image sensors may be performedfor the purpose of generating multiple images of the same scene inconnection with the HDR image generation. In some possible embodiments,an image, which may be created with a first field of a multi-field imagesensor, may be missing a component color in the full complement of acolor space used by the image processing system. In some possibleembodiments, a separate exposure may be made to transfer/shiftmeasurable imagery responses from a second field of the image sensor.The responses from the second field may be combined with the responsesin the first field to generate an image that comprises image data forthe full complement of the color space.

In some possible embodiments, instead of taking a separate exposure tocomplement a missing component color in a first image, a second imagewith the full complement of a color space may be used to construct oneor more histograms to predict pixel values (or portions of pixel values)for the missing component color in the first image. In some possibleembodiments, two or more histograms, which may correspond to differentsegments of an image, may be used to predict pixel values for themissing component color in each of these different segments.

In some possible embodiments, mechanisms as described herein form a partof a display system, including but not limited to a television, a laptopcomputer, netbook computer, cellular radiotelephone, electronic bookreader, point of sale terminal, desktop computer, computer workstation,computer kiosk, and various other kinds of terminals and display units.

Various modifications to the preferred embodiments and the genericprinciples and features described herein will be readily apparent tothose skilled in the art. Thus, the disclosure is not intended to belimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein.

2. IMAGE PROCESSING SYSTEM

FIG. 1A illustrates an example image processing system (100), inaccordance with some possible embodiments of the present invention. Insome possible embodiments, the image processing system (100) may beimplemented by one or more computing devices and may be configured withsoftware and/or hardware components that implement image processingtechniques for generating a wide dynamic range image based on at leasttwo relatively low dynamic range images.

In some possible embodiments, the system (100) may comprise an imagesensor (102), which may comprise light sensitive (or photoactive)elements configured to generate measurable signal responses to amountsof light illuminated on the light sensitive elements. For the purpose ofthe present invention, the memory space (106) may be any software and/orhardware component configured to generate measurable imagery responsesto the incoming light of scenes. In some possible embodiments, the imagesensor may be of a charge coupled device (CCD) type. An example ofmeasurable responses generated by light sensitive elements may be anelectrical charge packet produced as a response to an amount of lightilluminated on a light sensitive element of the image sensor (102). Thesystem (100), or the image sensor (102) therein, may be configured withmeans such as electrodes that may be used to provide necessary (e.g.,analog) waveforms to transfer electrical charge packets out of any ofthe light sensitive elements in the image sensor (102) tonon-photoactive storage elements such as electric charge registers.

The system (100), or the image sensor (102) therein, may be configuredwith software and/or hardware components to sample/convert amounts ofelectric charges in electrical charge packets generated from theincoming light of the scene into digital signals. The system (100) mayalso be configured with software and/or hardware components to processthe digital signals into pixel values that represent a part or whole ofan image of the scene, and to store the pixel values in memory space(106) of the image processing system (100). For the purpose of thepresent invention, the memory space (106) may be any software and/orhardware component (e.g., RAM, flash memory, etc.) configured to supportstoring, retrieving, and manipulating operations of digital informationsuch as pixel values.

The system (100) may comprise an image sensor controller (104)configured to operatively control the image sensor (102). For example,the image sensor controller (104) may comprise one or more ofelectronic, optical and/or mechanical components such as controllers,processors, sensors, flashes, lights, amplifiers, masks, mechanicalshutter, one or more lenses, mirrors, etc., to set/regulate an exposurefor capturing an image of the scene.

In some possible embodiments, the image sensor controller (104), or theimage sensor (102), may comprise a mechanical shutter (not shown)configured to support optical and/or mechanical shutter operations. Asused herein, an optical and/or mechanical shutter operation may refer toany operation that produces measurable imagery responses (e.g., electriccharge packets, etc.) for a particular time duration from the imagesensor (102), while blocking incoming light after the end of theparticular time duration to reach light sensitive elements of the imagesensor (102) at least for a second time duration.

In some possible embodiments, additionally and/or optionally, the imagesensor controller (104), or the image sensor (102), may comprise anelectronic shutter (110; as a part of the image sensor controller (104)for the purpose of illustration only) configured to support electronicshutter operations. As used herein, an electronic shutter operation mayrefer to any operation that transfers a part, or a whole, of measurableimagery responses (e.g., electric charge packets) for a particular timeduration from the image sensor (102) to cache, registers, or temporaryand/or permanent data stores while allowing the image sensor (102) to be(e.g., continually) exposed to incoming light of a scene after the endof the particular time duration to generate further measurable imageryresponses from at least some light sensitive elements of the imagesensor (102).

In some possible embodiments, additionally and/or optionally, the imagesensor controller (104), or the image sensor (102), may comprisesoftware and/or hardware configured to support image acquisitionoperations of various operational modes. Examples of operational modesmay include, but are not limited to, still image mode, programmaticallyoptimal mode, manual mode, video mode, preview mode, full resolutionmode, partial mode, histogram mode, etc.

The system (100) may comprise an image processor (108) configured toaccess/receive/manipulate partial or whole images of a scene, and toprocess these partial or whole images of a scene into a relatively widedynamic range image of the scene. In some possible embodiments, thepartial or whole images of the scene may be produced by the image sensor(102) with one or more electronic shutter operations. In some possibleembodiments, the partial or whole images of the scene may be produced bythe image sensor (102) under different operational modes.

3. LIGHT SENSITIVE AREA

FIG. 1B illustrates an example light sensitive area (112) of an imagesensor (e.g., 102 of FIG. 1A), in accordance with some possibleembodiments of the invention. The light sensitive area (112) maycomprise a plurality of pixels (e.g., 114-1, 114-2, etc.). For thepurpose of the present invention, any one of a number of different colorspaces may be used in an image processing system as described herein.For the purpose of illustration only, the image processing system (100)may use a red-green-blue (RGB) color space. In some possibleembodiments, each (e.g., 114-1) of the pixels may comprise one or morelight sensitive elements, arranged in one of one or more light sensitiveelement patterns; each of the light sensitive elements in a pixel (e.g.,114-1) may be configured to generate a response to light of certaincolors. For example, the pixel (114-1) may comprise four light sensitiveelements, in a pattern in which a first row comprises R and G and asecond row comprises G and B. In some possible embodiments, the lightsensitive element patterns in the pixels of the light sensitive area(112) may be created using a Bayer filter mosaic. In some possibleembodiments, light sensitive elements labeled as “G” in FIG. 1B may becovered by a green filter; light sensitive elements labeled as “R” inFIG. 1B may be covered by a red filter; and light sensitive elementslabeled as “B” in FIG. 1B may be covered by a blue filter. In someembodiments, a light sensitive element may also be responsive to one ormore other light colors other than a designated color. For example, a“G” light sensitive element in an “RG” row may be responsive to bluelight to a limited extent, while a “G” light sensitive element in a “GB”row may be responsive to red light to a limited extent. What lightcolors a specific light sensitive element in the light sensitive area(112) may depend on the structure and design of the pixel (e.g., asdetermined by the structure and design of the Bayer filter mosaicmentioned above) containing the specific light sensitive element.

In some possible embodiments, under techniques as described herein, animage processing system (e.g., 100 of FIG. 1A), or an image sensor(e.g., 102 of FIG. 1B), may comprise software and/or hardware configuredto determine a digital value of a measurable imagery response in each oflight sensitive elements in a pixel (e.g., 114-1), and to performconversion operations (including algebraic manipulations of differentdigital values of different light sensitive elements (RG and GB) of thepixel (114-1)) that generate a pixel value for the pixel (114-1). Insome possible embodiments, the pixel value may comprise component (orchannel) values for all components (or channels) of the color space usedin the image processing system (100). For example, in the RGB colorspace, a pixel value for a pixel (114-1) may comprise a component valuefor each of the red, green, and blue colors. In some possibleembodiments, pixel values generated for pixels of the light sensitivearea (112) may be within a relatively narrow dynamic range for luminance(which may be determined by the component values of R, G, and B in theRGB color space).

In some possible embodiments, an image sensor (e.g., 102 of FIG. 1A) maycomprise electric charge registers (not shown; which may be alignedalong the vertical direction of FIG. 1B) configured to receive electriccharge packets transferred/shifted out of light sensitive elements inthe light sensitive area (112) when a shutter operation including butnot limited to an electronic shutter operation is performed, forexample, through electrodes connected to the light sensitive elementsand to the electric charge registers. In some possible embodiments, theelectric charge registers and the light sensitive elements may beinterleaved in the light sensitive area (112). In some possibleembodiments, each different vertical column (e.g., 116-1) of lightsensitive elements may correspond to a different vertical column ofelectric charge registers. In some possible embodiments, two or morevertical columns (e.g. 116-1, 116-2, 116-3, etc.) of light sensitiveelements may correspond to a single vertical column of electric chargeregisters. In some possible embodiments, the image sensor (102), or thelight sensitive area (112) therein, may comprise software and hardwarecomponents to transfer/shift electric charge packets out of the electriccharge registers (e.g., along a horizontal direction of FIG. 1B), and tosample/convert the electric charge packets into digital image data of ascene.

In some possible embodiments, an image sensor (e.g., 102 of FIG. 1A) mayoptionally and/or alternatively employ multi-field readout techniques.In some possible embodiments, light sensitive elements in the imagesensor (102) may be grouped into a plurality of (e.g., interleaved)light sensitive fields. As used herein, different light sensitive fieldsmay not occupy different exclusive spatially contiguous fields but mayinterleave with one another spatially. In some possible embodiments,different light sensitive fields may share a common set of electrodelines. Thus, an image sensor implementing multi-field readout techniquesmay have fewer electrode lines, and may be able to engineer more lightsensitive elements in a unit area than otherwise. In some possibleembodiments, multi-field readout techniques as described herein may, butare not limited to one or more of, support reading out measurableimagery responses from the light sensitive fields in a sequential order,in a round-robin, in a different order including repeatedly reading outone or more of the light sensitive fields, etc.

In some possible embodiments, measurable imagery responses of lightsensitive elements in the plurality of light sensitive fields may beshifted out of the light sensitive elements one light sensitive field ata time. In some possible embodiments, measurable imagery responses oflight sensitive elements in a light sensitive field may be shifted outof the light sensitive elements simultaneously.

In an example, light sensitive elements in the image sensor (102) may begrouped into two light sensitive fields. A first vertical column (e.g.,116-1 of FIG. 1B) of light sensitive elements may belong to a firstlight sensitive field; a second vertical column (e.g., 116-2 of FIG. 1B)of light sensitive elements may belong to a second light sensitivefield; a third vertical column (e.g., 116-3 of FIG. 1B) of lightsensitive elements may belong to the first light sensitive field; and soon.

In another example, light sensitive elements in the image sensor (102)may be grouped into three light sensitive fields. A first verticalcolumn (e.g., 116-1 of FIG. 1B) of light sensitive elements may belongto a first light sensitive field; a second vertical column (e.g., 116-2of FIG. 1B) of light sensitive elements may belong to a second lightsensitive field; a third vertical column (e.g., 116-3 of FIG. 1B) oflight sensitive elements may belong to a third light sensitive field;and so on.

In various possible embodiments, the number of light sensitive fields inan image sensor herein may be a finite positive number greater than onesuch as two, three, four, five, six, seven, eight, etc.

4. OPERATIONS FOR CAPTURING MULTIPLE IMAGES

FIG. 2A illustrates example operations of an image processing system(e.g., 100) for capturing multiple images of a scene, in accordance withsome possible embodiments of the present invention. A mechanical shutterwith an image sensor (e.g., 102 of FIG. 1A) of the image processingsystem (100) may be opened at time 202-1 and closed at time 202-2 for atotal time duration 206. In some possible embodiments, a number ofoptical, electronic, mechanical settings such as aperture settings, atime duration for the mechanical shutter to remain open, one or morefocal lengths of one or more lenses, etc., may be set by an image sensorcontroller (104) of the image processing system (100) for the purpose ofcapturing the multiple images. Some of these settings may be caused orinfluenced by one or more user inputs provided to the image processingsystem (100) and/or by environment conditions (e.g., ambient lightlevels, subject light levels in the scene, etc.) detected/received bythe image processing system (100). At time 202-1, measurable imageryresponses from previous images if any may be cleared. An exposure timermay be used to count down to the time 202-2 at which the mechanicalshutter may be closed.

At time 204, the image sensor controller (104), or an electronic shutter(e.g., 110 of FIG. 1A) therein, may transfer/shift first measurableimagery responses out of a part, or a whole, of light sensitive elementsin the image sensor (102) to non-photoactive storage elements such aselectric charge registers for (e.g., temporary or transitional) storagepurposes.

Shifting the first measurable imagery responses as described abovesimultaneously ends a first exposure 208-1 that generates a first imageof the scene and begins a second exposure 208-2 that generates a secondimage of the scene. At time 202-2, when the mechanical shutter closes,the second exposure 208-2, which generates the second image of thescene, ends. In some possible embodiments, the first measurable imageryresponses may be read out of the non-photoactive storage elements whilethe second exposure 208-2 is in progress. In some possible embodiments,second measurable imagery responses accumulated during the secondexposure 208-2 may be read out of the light sensitive element to thenon-photoactive storage elements after the first measurable imageryresponses from the first image have been moved and cleared out of thenon-photoactive storage elements.

In some possible embodiments, the first image may be generated based onthe first measurable imagery responses. In some possible embodiments,the first image may comprise information derived from the firstmeasurable imagery responses. In some possible embodiments, the secondimage of the scene may be generated based on both the first measurableimagery responses and the second measurable imagery responses. In someother possible embodiments, the second image of the scene may begenerated based on the second measurable imagery responses.

In a single field imager sensor, the first measurable imagery responsesmay comprise responses from an entire light sensitive area (e.g., 112 ofFIG. 1B) for the first exposure 208-1; the second measurable imageryresponses may comprise responses from the same entire light sensitivearea (e.g., 112 of FIG. 1B) for the second exposure 208-2. The firstmeasurable imagery responses, as well as the second measurable imageryresponses, may be used to generate a full resolution image of the scenewith all component colors.

In a multi-field image sensor, the first measurable imagery responsesmay comprise only responses from a first field of a light sensitive area(e.g., column 116-1 in light sensitive area 112 of FIG. 1B) accumulatedfor the first exposure 208-1; the second measurable imagery responsesmay comprise responses from the entire light sensitive area (e.g., 112of FIG. 1B). In some possible embodiments, the second measurable imageryresponses in the first field may comprise responses accumulated for thesecond exposure 208-2 only since the responses in the first fieldaccumulated for the first exposure 208-1 have already beentransferred/shifted out, while the second measurable imagery responsesin other fields (or the other field in the case of a two-field imagesensor) may comprise responses accumulated for both the first exposure208-1 and the second exposure 208-2.

In some possible embodiments, algebraic operations such as aggregationsmay be performed for the responses from the light sensitive elements inthe first field so that the second measurable imagery responses in thefirst field that are used in part to generate the second image maycomprise responses accumulated for both the first exposure 208-1 and thesecond exposure 208-2.

In some other possible embodiments, algebraic operations such asaggregations may not be performed for the responses from the lightsensitive elements in the first field. Instead, since the secondmeasurable imagery responses in the first field and in other fields areobtained from different time durations respectively (e.g., the secondexposure 208-2 for the first field and the first and second exposures208-1 and 208-2 for the other fields), interpolation operations may beperformed to correlate the second measurable imagery responses in thefirst field and the other fields. As used herein, an interpolationoperation of two measurable imagery responses may refer to one or more(e.g., algebraic, table-driven, etc.) manipulations of one or both ofthe two corresponding measurable imagery responses based on one or morerelationships between the two corresponding measurable imageryresponses; the relationships may be based on exposure time durationsrelated to the measurable imagery responses and/or physicalcharacteristics of the light sensitive elements that generate themeasurable imagery responses, etc. As a result of the interpolationoperations, the second image may be generated comprising informationderived from the second measurable imagery responses only.

In some possible embodiments, the first image and the second image, ofthe same scene, may be outputted by the image sensor (102) and may bestored in memory space (e.g., 106) of the image processing system (100)for further processing.

In some possible embodiments in which a light sensitive area (e.g., 112)of the image sensor (102) comprises a single light sensitive field, thefirst image may comprise measurable imagery responses from eachindividual light sensitive element (e.g., light sensitive elements incolumns 116-1 and 116-2, 116-3, etc., of FIG. 1B) of the light sensitivearea (112). In some possible embodiments in which a light sensitive area(e.g., 112) of the image sensor (102) comprises two light sensitivefields, the first image may comprise measurable imagery responses fromlight sensitive elements (e.g., light sensitive elements in column 116-1and 116-3, but not in column 116-2, of FIG. 1B) in one half of the lightsensitive area (112). In some possible embodiments in which a lightsensitive area (e.g., 112) of the image sensor (102) comprises threelight sensitive fields, the first image may comprise measurable imageryresponses from light sensitive elements (e.g., light sensitive elementsin column 116-1, but not in columns 116-2 and 116-3, of FIG. 1B) in athird of the light sensitive area (112). Thus, in some possibleembodiments in which a light sensitive area (e.g., 112) of the imagesensor (102) comprises N (a positive integer) light sensitive fields,the first image may comprise measurable imagery responses from lightsensitive elements in a particular light sensitive field constitutingone Nth of the light sensitive area (112).

In some possible embodiments in which the image processing system (100)uses a color space that comprises three or more component colors (e.g.,red, green, and blue in a RGB color space), one component color in thecolor space may be missing from the first image. For example, in thoseembodiments in which a light sensitive area (e.g., 112) of the imagesensor (102) comprises an even positive number of light sensitivefields, the first image may comprise measurable imagery responses fromall of red light sensitive elements and one half of the green lightsensitive elements, but not blue light sensitive elements, of the lightsensitive area (112). It may be noted that in embodiments in which atwo-light-sensitive-fields image sensor is used, the first imagegenerated with techniques herein may still contain per pixel imagerydata (e.g., only red and green information of an individual pixel), andhence may be of the same spatial resolution as that of the second image,even though a component color is missing in the first image. However, inother embodiments in which a four-light-sensitive-fields image sensor ora six-light-sensitive-fields image sensor (or still higher even numberfields) is used, the first image generated with techniques herein maynot contain per pixel imagery data (e.g., only red and green informationof selected but not all pixels), and hence may be of a lower spatialresolution; in the meantime, a component color is missing in the firstimage.

On the other hand, in embodiments in which a light sensitive area (e.g.,112) of the image sensor (102) comprises an odd positive number of lightsensitive fields, the first image may comprise measurable imageryresponses from light sensitive elements of the light sensitive area(112) in all component colors. In a particular embodiment of theseembodiments, light sensitive elements of different component colors frommore than one pixel may be combined to generate a composite pixel. Thus,in embodiments in which a three-light-sensitive-fields image sensor or afive-light-sensitive-fields image sensor (or still higher odd numberfields) is used, the first image generated with techniques herein maynot contain per pixel imagery data (e.g., all component colorinformation of selected but not all pixels), and hence may be of a lowerspatial resolution.

FIG. 2C illustrates an example timeline of capturing an HDR image usinga multi-field image sensor. In an example embodiment, a mechanicalshutter is open from time t₀ to time t_(L). From time t₀ to time t_(S)is the total time for capturing a short exposure image. Theshort-exposure image is shifted to an image buffer (e.g.,non-photoactive storage elements) from time to time. For example, aftertime t₁ and until time t₂, the first field is shifted out of the imagesensor into the image buffer. After time t₂ and until time t₃, thesecond field is shifted out of the image sensor into the image buffer.Shifting of fields of the image sensor continues while the shutter isopen and until t_(S). The collection of these fields becomes a shortexposure image, which may include all primary colors in someembodiments. After time t_(L), all light sensitive fields are shiftedfor image sensory responses accumulated up to time t_(L) (e.g., when themechanical shutter is closed), thereby creating a long exposure image.The short exposure image and the long exposure image are then combinedinto an HDR image under techniques as described herein.

In an example embodiment, a first field of an image sensor is capturedwithout all primary colors in a color space. FIG. 2D illustrates afurther example timeline of capturing an HDR image using a multi-fieldimage sensor. A mechanical shutter is open from time t₀₀ to time t_(L0).From time t₀₀ to time t_(s0) is a short exposure. After time t_(s0), afirst field of the image sensor is shift out. After time t_(L0), whichis the end of a long exposure, the remainder of the first field(captured between time t_(S0) and time t_(L0)) and the remaining fieldsin the image sensor, captured from time t₀₀ to time t_(L0), are shiftedout. At this point, an image of the short exposure is obtained, whichmay be with missing primary colors. In addition, another image of thelong exposure is also obtained, which may be with all primary colors. Insome possible embodiments in which one of the obtained images is withmissing primary colors, the missing primary colors are estimated,interpolated, or otherwise predicted.

In an example, after time t_(L0), an image processing system hereintakes another short exposure to obtain a second short exposure image,e.g., from time t₀₁ to time t_(s1). This second exposure image has allthe primary colors in the color space. Color ratios obtained with thesecond exposure image are used to interpolate the missing primary colorsin the previous short exposure image. Alternatively and/or additionally,if the long exposure image was taken in quick succession after aprevious picture, the color ratios of the previous picture to estimatethe missing primary colors.

In another example, a library of HDR pictures (preferably taken by thesame camera) is used to create a two-dimensional histogram of colorratios. This histogram is used to estimate or predict the missingprimary colors.

5. MIXED OPERATIONAL MODES FOR CAPTURING MULTIPLE IMAGES

FIG. 3A illustrates example operations of an image processing system(e.g., 100) for capturing multiple images of a scene, in accordance withsome possible embodiments of the present invention. A mechanical shutterwith an image sensor (e.g., 102 of FIG. 1A) of the image processingsystem (100) may be opened at time 302-1 and closed at time 302-2 for anexposure time duration 308-1. In some possible embodiments, an imagesensor controller (104) of the image processing system (100) may set afirst operational mode for the exposure time duration 308-1. Some ofoptical, electronic, or mechanical settings of the image processingsystem (100) in the first operational mode may be caused or influencedby one or more user inputs provided to the image processing system (100)and/or by environment conditions (e.g., ambient light levels, subjectlight levels in the scene, etc.) detected/received by the imageprocessing system (100). At time 302-1, measurable imagery responsesfrom previous images if any may be cleared. An exposure timer may beused to count down to the time 302-2 at which the mechanical shutter maybe closed. At time 302-2, the image sensor controller (104) maytransfer/shift first measurable imagery responses out of light sensitiveelements in the image sensor (102) to non-photoactive storage elementssuch as electric charge registers for (e.g., temporary or transitional)storage purposes.

Shifting the first measurable imagery responses as described abovesimultaneously ends a first exposure 308-1 that generates a first imageof the scene. In some possible embodiments, a second exposure 308-2 thatwill generate a second image of the scene begins at time 302-3.

In some possible embodiments, the image sensor controller (104) may seta second different operational mode for the exposure time duration308-2. Some of optical, electronic, or mechanical settings of the imageprocessing system (100) in the second operational mode may be caused orinfluenced by one or more user inputs provided to the image processingsystem (100) and/or by environment conditions (e.g., ambient lightlevels, subject light levels in the scene, etc.) detected/received bythe image processing system (100).

At time 302-4, when the mechanical shutter closes, the second exposure308-2 ends. In some possible embodiments, the first measurable imageryresponses may be read out of the non-photoactive storage elements whilethe second exposure 308-2 is in progress. In some possible embodiments,second measurable imagery responses accumulated during the secondexposure 208-2 may be read out of the light sensitive element to thenon-photoactive storage elements after the first measurable imageryresponses from the first image have been moved and cleared out of thenon-photoactive storage elements.

In some possible embodiments, the first image may be generated based onthe first measurable imagery responses. In some possible embodiments,the second image of the scene may be generated based on both the firstmeasurable imagery responses and the second measurable imageryresponses. Since the first measurable imagery responses and the secondmeasurable imagery responses are obtained from different exposures(e.g., the first and second exposures 308-1 and 308-2), interpolationoperations may be performed to correlated the first measurable imageryresponses and the second measurable imagery responses. In some otherpossible embodiments, the second image of the scene may be generatedbased on only the second measurable imagery responses (e.g., not basedon the first measurable imagery responses).

In some possible embodiments, the first image and the second image, ofthe same scene, may be outputted by the image sensor (102) and may bestored in memory space (e.g., 106) of the image processing system (100)for further processing.

In some possible embodiments, at least one of the first image and thesecond image as discussed above may be a high resolution image (e.g., astill photographic image with a still image operational mode or a highresolution operational mode). In some possible embodiments, at least oneof the first image and the second image may be a low resolution image(e.g., a video frame with a video operational mode or a low resolutionoperational mode). In some possible embodiments, at least one of thefirst image and the second image may comprise data for all componentcolors of a color space (e.g., red, blue, and green in the RGB colorspace). In some possible embodiments, at least one of the first imageand the second image may comprise data for some but not all of componentcolors in the color space (e.g., red and green, but not blue in the RGBcolor space).

For the purpose of illustration, it has been described that the firstexposure 308-1 may be ended with the closing of the mechanical shutter.It should be noted that for the purpose of the invention, a firstexposure herein may be ended with an electronic shutter operation. Forexample, instead of closing the mechanical shutter at time 302-2 at theend of the first exposure and opening the mechanical shutter at time302-3 at the start of the second exposure, an electronic shutteroperation may be used to end the first exposure and to begin the secondexposure. In some possible embodiments, even when an electronic shutteroperation is used, the operational modes of the first exposure and thesecond exposure may be different. Therefore, these and other ways ofcapturing multiple images of a scene with different operational modesmay be used for the purpose of the present invention.

In some possible embodiments, the time interval between the end (time302-2) of the first exposure 308-1 and the start (time 302-3) of thesecond exposure 308-2 may be relatively short, and may even be zero orclose to zero (e.g., when an electronic shutter operation is implementedto end the first exposure 308-1 and to start the second exposure 308-2).

6. MIXED OPERATIONAL MODES IN LIGHT SENSITIVE AREA

FIG. 3B illustrates an example light sensitive area (312) that may beused by an image processing system (e.g., 100 of FIG. 1A) to supportcapturing multiple images of a scene in different operational modes, inaccordance with some possible embodiments of the invention. In somepossible embodiments, the light sensitive area (312) may be the same as,or similar to, the light sensitive area (112) of FIG. 1B.

In some possible embodiments, when the image processing system (100) isset to a specific high resolution mode (e.g., a still image operationalmode), one or more light sensitive elements in a pixel (e.g., 314-1) ofthe light sensitive area (312) may be used to produce image data for onepixel in an output image (e.g., the first image discussed above inconnection with FIG. 3A). In some possible embodiments, when the imageprocessing system (100) is set to a specific low resolution mode (e.g.,a video operational mode), one or more light sensitive elements in aplurality of pixels (e.g., 314-1 through 314-4) of the light sensitivearea (312) may be used to produce image data for one pixel in an outputimage (e.g., the second image discussed above in connection with FIG.3A). For the purpose of the present invention, summing (e.g., summingelectric charges from two or more light sensitive elements of the sametype), averaging, mean value computing, sub-sampling, Gaussianfiltering, non-Gaussian filtering, etc., or combinations of some of theforegoing, performed in an analog or digital representation, may be usedto generate image data for a pixel in the output image from a pluralityof pixels (314-1 through 314-4) of the light sensitive area (312) in anoperational mode (e.g., a video mode, a preview mode, or any operationalmode in which the spatial resolution is below a full resolutionsupported by the imaging processing system (100)).

For the purpose of illustration only, it has been described that fourpixels of an image sensitive area may be used to generate image data fora pixel of an output image. For the purpose of the present invention, adifferent plurality of, for example, 8, 16, etc., pixels of an imagesensitive area may be used to generate image data for a pixel of anoutput image.

7. ADDITIONAL EXPOSURE TO COMPLEMENT MISSING COMPONENT COLOR

FIG. 2B illustrates example operations of an image processing system(e.g., 100) for capturing multiple images of a scene, in accordance withsome possible embodiments of the present invention. The mechanicalshutter with an image sensor (e.g., 102 of FIG. 1) of the imageprocessing system (100) may be opened at time 202-3 and closed at time202-4 for a total time duration 206-1. In some possible embodiments, anumber of optical, electronic, mechanical settings such as aperturesettings, a time duration for the mechanical shutter to remain open, oneor more focal lengths of one or more lenses, etc., may be set by animage sensor controller (104) of the image processing system (100) forthe purpose of capturing the multiple images. Some of these settings maybe caused or influenced by one or more user inputs provided to the imageprocessing system (100) and/or by environmental conditions (e.g.,ambient light levels, subject light levels in the scene, etc.)detected/received by the image processing system (100). At time 202-3,measurable imagery responses from previous images, if any, may becleared. An exposure timer may be used to count down to the time 202-4at which the mechanical shutter may be closed.

At time 204-1, the image sensor controller (104), or an electronicshutter (e.g., 110 of FIG. 1A) therein, may transfer/shift firstmeasurable imagery responses out of a part, or a whole, of lightsensitive elements in the image sensor (102) to non-photoactive storageelements such as electric charge registers for (e.g., temporary ortransitional) storage purposes. Shifting the first measurable imageryresponses as described above simultaneously ends a first exposure 208-3that generates the first measurable imagery responses of the scene andbegins a second exposure 208-4 that will generate second measurableimagery responses of the scene.

At time 204-2, the image sensor controller (104), or the electronicshutter (110) therein, may transfer/shift the second measurable imageryresponses out of a part, or a whole, of the light sensitive elements inthe image sensor (102) to the non-photoactive storage elements for(e.g., temporary or transitional) storage purposes. In some possibleembodiments, the first measurable imagery responses may be read out ofthe non-photoactive storage elements while the second exposure 208-4 isin progress. In some possible embodiments, second measurable imageryresponses accumulated during the second exposure 208-4 may be read outof the light sensitive element to the non-photoactive storage elementsafter the first measurable imagery responses have been moved and clearedout of the non-photoactive storage elements. Shifting the secondmeasurable imagery responses as described above simultaneously ends asecond exposure 208-4 that generates the second measurable imageryresponses of the scene and begins a third exposure 208-5 that willgenerate third measurable imagery responses of the scene.

At time 202-4, when the mechanical shutter closes, the third exposure208-5, which generates the third measurable imagery responses of thescene, ends. In some possible embodiments, the second measurable imageryresponses may be read out of the non-photoactive storage elements whilethe third exposure 208-5 is in progress. In some possible embodiments,third measurable imagery responses accumulated during the secondexposure 208-5 may be read out of the light sensitive element to thenon-photoactive storage elements after the second measurable imageryresponses have been moved and cleared out of the non-photoactive storageelements.

In some possible embodiments, two images may be generated from thefirst, second, and third measurable imagery responses. In some possibleembodiments, a first image may be generated based on the first andsecond measurable imagery responses. In some possible embodiments, asecond image of the scene may be generated based on the third measurableimagery responses. In some other possible embodiments, the second imageof the scene may be generated based on the third measurable imageryresponses and either or both of the first measurable imagery responsesand the second measurable imagery responses.

In a single field image sensor, each of the first measurable imageryresponses, the second measurable imagery responses, and the thirdmeasurable imagery responses may comprise responses from an entire lightsensitive area (e.g., 112 of FIG. 1B), albeit for possibly differenttime durations depending on the first, second and third exposures 208-3,208-4, and 208-5. Each of the first measurable imagery responses, thesecond measurable imagery responses, and the third measurable imageryresponses may be used to generate a full resolution image of the scenewith all component colors.

In a multi-field image sensor, the first measurable imagery responsesmay comprise only responses from a first field of a light sensitive area(e.g., column 116-1 in light sensitive area 112 of FIG. 1B) accumulatedfor the first exposure 208-1, while the second measurable imageryresponses may comprise responses from a second field of the lightsensitive area (e.g., column 116-2 in light sensitive area 112 of FIG.1B). In some possible embodiments, the first measurable imageryresponses in the first field may comprise responses accumulated for thefirst exposure 208-3, while the second measurable imagery responses inthe second field may comprise responses accumulated for the first andsecond exposures 208-3 and 208-4. The third measurable imagery responsesin the first field may comprise responses accumulated for the second andthird exposures 208-4 and 208-5, since the responses in the first fieldaccumulated for the first exposure 208-3 have already beentransferred/shifted out. The third measurable imagery responses in thesecond field may comprise responses accumulated for the third exposure208-5, since the responses in the second field accumulated for the firstand second exposures 208-3 and 208-4 have already beentransferred/shifted out. In some possible embodiments, the thirdmeasurable imagery responses in other fields if any (e.g., a four-fieldimage sensor) may comprise responses accumulated for the first, secondand third exposures 208-3, 208-4 and 208-5.

In some possible embodiments, algebraic operations such as aggregationsmay be performed for the responses from the light sensitive elements inthe first or second field so that the responses in the first or secondfield that are used at least in part to generate one of the first andsecond images may be made the same in exposure time durations.

In some possible embodiments, since the third measurable imageryresponses in the first field, in the second field, and in other fieldsif any, are obtained from different time durations respectively (e.g.,the second and third exposures 208-4 and 208-5 for the first field, thethird exposure 208-5 for the second field, and the first, second andthird exposures 208-3, 208-4 and 208-5 for the other fields if any),interpolation operations may be performed to correlate the thirdmeasurable imagery responses in the first field, the second field, andthe other fields.

Thus, in some possible embodiments, since different measurable imageryresponses may be obtained from different time durations, interpolationoperations may be performed to correlate the different measurableimagery responses for normalization or comparison purposes.

In some possible embodiments, the first image and the second image, ofthe same scene, may be outputted by the image sensor (102) and may bestored in memory space (e.g., 106) of the image processing system (100)for further processing.

In some possible embodiments in which the image processing system (100)uses a color space that comprises three or more component colors (e.g.,red, green, and blue in a RGB color space), one component color in thecolor space may be missing from the first measurable imagery responses.For example, in those embodiments in which a light sensitive area (e.g.,112) of the image sensor (102) comprises an even positive number oflight sensitive fields, the first measurable imagery responses maycomprise measurable imagery responses from all of red light sensitiveelements and one half of the green light sensitive elements, but notblue light sensitive elements, of the light sensitive area (112); thesecond measurable imagery responses may comprise measurable imageryresponses from all of blue light sensitive elements and the other halfof the green light sensitive elements, but not red light sensitiveelements, of the light sensitive area (112). By combining at least thefirst and second measurable imagery responses into both of the first andsecond images may make sure that both images comprise image data for allcomponent colors of the color space used in the image processing system(100).

It has been shown that techniques as described herein may be used tocreate two images with different spatial resolution and/or differentcolor content. Since the images may be produced with differentexposures, the images of the same scene with different luminance levelsand/or different spatial resolutions and/or different color contents maybe generated under the techniques. In some possible embodiments, thesame techniques may be used to generate more than two images of the samescene with different luminance levels and/or different spatialresolutions and/or different color contents.

8. PREDICTING MISSING COMPONENT COLOR

In various possible embodiments, instead of using measurable imageryresponses from multiple exposures to create an image (e.g., the firstimage discussed in connection with FIG. 2B), an image generated undertechniques herein may or may not be missing one or more componentcolors. In some possible embodiments in which an image is missing acomponent color, instead of using measurable imagery responses (e.g.,the second measurable imagery responses discussed in connection withFIG. 2B) to complement the missing color in the image, pixel values forthe missing component color in the image may be predicted.

In some possible embodiments, one or more histograms may be used topredict a pixel value for the missing component color (e.g., the bluepart of a RGB pixel value if blue is the missing component color). Theone or more histograms may be generated in one or more of variousdifferent methods. In an example, a histogram as described herein may becomputed using pixel values in a reference image of the same scene,wherein the pixel values in the reference image may comprise allcomponent colors. The reference image as described herein for generatingthe histograms may be any image that does not have a missing componentcolor and be of the same scene. In some possible embodiments, thereference image may comprise pixel values at least in part throughinterpolation and/or extrapolation and/or other image processingoperations. In some possible embodiments, the reference image may beobtained from a pre-shot of the scene, a post-shot of the scene, aphotometric shot of the scene, a view finder type of shot of the scene,etc.

In some possible embodiments, a histogram here may represent arelationship among all component colors (which may number to three,four, five, six, etc. in various possible embodiments) of a color spacesuch that, given pixel values for two or more component colors of thecolor space, pixel values for one or more other (missing) componentcolors may be determined. In some possible embodiments, a histogram asdescribed herein may be implemented as one or more pixel value functionseach of which has pixel values of the two or more component colors asindependent variables; the one or more pixel value functions generatepixel values for the one or more missing component colors when the pixelvalues of the two or more component colors are known. In some possibleembodiments, the pixel values of the two or more component colors may beprovided by an image for which the pixel values for the missingcomponent colors are to be predicted with one or more histograms asdescribed herein.

For the purpose of illustration, it has been described that a referenceimage used to generate at least in part a histogram herein may be of thesame scene as that of an image for which pixel values for one or morecomponent colors are to be predicted with the histogram. It should benoted that the present invention is not so limited. In an example,instead of using a reference image of the same scene, other referenceimages may be used. For example, one or more reference images of thesame or different scenes may be used to construct a histogram asdescribed herein. None, some, or all of these reference images may befrom an image collection. In some possible embodiments, the imagecollection may be preconfigured on or off an image processing system(e.g., 100) herein. In another example, one or more black-body radiationspectrums comprising relationships of light wavelengths at one or moregiven temperature or other factors may be used to predict/estimate othercolors given some known colors. In some possible embodiments, onlyhistograms computed may be configured on an image processing system(e.g., 100), but not reference images or an image collection orblack-body radiation spectrums. Therefore, these and other ways ofconstructing histograms for the purpose of predicting one or moremissing component colors may be used for the purpose of the presentinvention.

In some possible embodiments, one of one or more images generated usingtechniques herein may be of a lower spatial resolution than that ofanother of the one or more images. In some possible embodiments, theimage of the lower spatial resolution may comprise a lower number ofpixels than the image of a higher spatial resolution. In some possibleembodiments, the image of the lower spatial resolution may be upsampled.For example, an empty stack frame comprising the same number of pixelsas the image of the higher spatial resolution may be used to containpixel values upsampled from the image of the lower spatial resolution.Upsampling may scale up in the spatial dimensions, but not in thespatial resolution. In some possible embodiments, a pixel value in theimage of the lower spatial resolution may be used to set a plurality ofpixels (as determined by a ratio of the number of pixels in the image ofthe higher resolution versus the number of pixels in the image of thelower resolution) in the stack frame.

In some possible embodiments, multiple images as generated from an imageprocessing system (e.g., 100) of the same scene may be used tocreate/generate an HDR image of the scene. In some possible embodiments,one or more images herein may comprise image data affected by smearing,for example, caused by an electronic shutter operation thattransfers/shifts measurable imagery responses from light sensitiveelements while continually exposing the light sensitive elements toincoming light. In some embodiments, such smearing may be compensatedfor by comparing image data for a first image, which may comprise imagedata transferred/shifted out while the mechanical shutter remains open,to image data for a second image and/or a third image, which may betransferred/shifted out after the mechanical shutter has been closed. Insome possible embodiments, data corresponding to columns from the firstimage that are in shadow, and therefore not affected too much bysmearing, are selected and data from those columns is used to gaindetail in shadow portions of an HDR image that is obtained by combiningimage data from the second image and/or the third image. In somepossible embodiments, the image data from the first image may be used tocontribute to the HDR image without compensating for smearing. In somesuch embodiments, data from the first image may be weighted differentlyin creating the HDR image than data from the second image and/or thethird image.

In some possible embodiments, multiple images generated under techniquesas described herein may have at least two different exposures. Forexample, one of the multiple images may have a longer exposure thananother of the multiple images. In some possible embodiments, an imageprocessing system (e.g., 100) may look through an image (e.g., a fullspatial resolution image) to locate overexposed and/or underexposedpixels. The pixel values for these overexposed and/or underexposedpixels may be corrected with pixel values for the same pixels from oneor more different images. In some possible embodiments, the one or moredifferent images may comprise image data for all component colors of acolor space in effect. In various possible embodiments, any of the oneor more different images herein may comprise predicted pixel values,interpolated and/or extrapolated pixel values, or pixel valuescomplemented by measurable imagery responses from one or more differentexposures.

In some possible embodiments, the multiple images that are used togenerate an HDR image may not necessarily be from a single mechanicalshutter operation. In some possible embodiments, the multiple imagesthat are used to generate an HDR image may not necessarily be from asingle electronic shutter operation.

In some possible embodiments, histograms as described herein may besegmented. For example, a histogram may be used to predict missing colorcomponents for a spatial segment of an image, rather than all spatialportions of the image. For example, a bottom segment of the image mayuse a histogram that is different from one used in a top segment of theimage. In some embodiments, one or more reference images may be dividedinto multiple segments (possibly overlapping). Pixel values from each ofthe multiple segments may be used to determine at least a segmentedhistogram as described herein.

In an example embodiment, a set of HDR images is used to generate ajoint histogram that relates color channel ratios. FIG. 6 illustratesexample histogram bins derived from a set of 436 HDR images. In FIG. 6,the brightness of each patch represents population frequency. One orboth of Red-Green and Blue-Green scales in FIG. 6 may be used todetermine a missing primary color (e.g., blue, red, etc.) using a ratio(e.g., between red and green, between blue and green. The ratio may beused to determine which histogram line (e.g., row or column) to use indetermining the missing primary color.

In an example, an image contains a pixel with a missing blue color. Thered and green channels of the pixel are used to compute a Red-Greenratio for the pixel. If the ratio is very small, the top row of thejoint histogram of FIG. 6 is used to find the mean value for theBlue-Green ratio based on that row's statistics. The mean value may beused to predict the missing primary color (blue). This simple techniqueperforms surprisingly well, especially when the statistics for thehistogram are gathered from similar images. A basic enhancement of onlytaking statistics for the highlights in the HDR images may be followedin some possible embodiments in which the histogram is applied to shortexposures with missing primary colors. On the other hand, a basicenhancement of only taking statistics for the shadows in the HDR imagesmay be followed in some possible embodiments in which the histogram isapplied to long exposures with missing primary colors.

Other optimizations may also be used, such as using histogram data fromanother exposure to decide what to do with an image with a missingprimary color. A mixture of statistics gathered over many images withstatistics including images from the current scene may be used and maybe proved especially valuable, in tie-breaking or deciding cases wherethe histogram has multiple peaks (e.g., bimodal regions).

In an example, a set of HDR images is used to determine a jointhistogram of red ratios and blue ratios, defined as:red_ratio=(red−grn)/(red+grn)blu_ratio=(blu−grn)/(blu+grn)

The values of the ratios go from −1 to 1. In an example embodiments,only pixels whose luminance is above a certain threshold (e.g., 1 EVabove the linear average luminance) are used so as to relate to thebrighter portions of the image with a missing primary color, as expectedfrom a shorter exposure.

The histogram built under techniques herein is used to determine themost likely color value for a missing primary color, given one or moreratios between the non-missing primary colors. For example, if a pixelis with a missing red color, but the blu_ratio is known to be 0.3, themean value (or mode) for the red_ratio at the blu_ratio is assigned forthe purpose of estimating or predicting a value for the red channel. Thevalue for the red channel may be computed as follows:red=grn*(1+red_ratio)/(1−red_ratio)wherein red_ratio is assumed to be less than one (1).

Similarly, a value for the blue channel may be estimated or predictedbased on the histogram, given a red_ratio.

9. EXAMPLE PROCESS FLOW

FIG. 4A illustrates an example process flow according to a possibleembodiment of the present invention. In some possible embodiments, oneor more computing devices or components such as an image processingsystem (e.g., 100) may perform this process flow. In block 402, theimage processing system (100) may generate a first image of a scene witha first operational mode (e.g., one of still image mode,programmatically optimal mode, manual mode, video mode, preview mode,full resolution mode, partial mode, histogram mode, etc.) of the imageprocessing system.

In block 404, the image processing system (100) may generate a secondimage of the scene with a second different operational mode (e.g., adifferent one of still image mode, programmatically optimal mode, manualmode, video mode, preview mode, full resolution mode, partial mode,histogram mode, etc.) of the image processing system. The first imagemay be of a first spatial resolution, while the second image may be of asecond spatial resolution. In a possible embodiment, the first spatialresolution may be higher than the second spatial resolution.

In block 406, the image processing system (100) may combine the firstimage and the second image into an output image of the scene. The outputimage may be of a higher dynamic range than either of the first imageand the second image and may be of a spatial resolution higher than thesecond spatial resolution.

In some possible embodiments, the first operational mode may be a fullresolution operational mode of the image processing system, while thesecond operational mode may be a low resolution mode, in relation to thefirst operational mode, of the image processing system. In some otherpossible embodiments, the first operational mode may not be a fullresolution operational mode of the image processing system, while thesecond operational mode may be a high resolution mode, in relation tothe first operational mode, of the image processing system.

In some possible embodiments, the first image may be generated with afirst exposure, while the second image may be generated with a secondexposure. In a particular possible embodiment, the first image may bemore exposed than the second image. In another particular possibleembodiment, the first image may be less exposed than the second image.

In some possible embodiments, the first image may comprise first pixelvalues for a first plurality of subsets of pixels, whereas the secondimage may comprise second pixel values for a second plurality of pixels.Here, each different individual subset of pixels in the first image hasa one-to-one correspondence relationship with a different individualpixel of the second image.

In some possible embodiments, the image processing system (100) mayupsample the second image to a stack frame that comprises a same numberof pixels as the first image so the first and second images may becombined into the output image.

In some possible embodiments, the output image may be of a spatialresolution the same as the higher of the first and second images. Insome possible embodiments, the output image may be of a spatialresolution between those of the first and second images. In somepossible embodiments, a first pixel in the output image may have a firstluminance value derived from one or more first luminance values for oneor more first pixels in the first image, while a second pixel in theoutput image may have a second luminance value derived from one or moresecond luminance values for one or more second pixels in the secondimage.

FIG. 4B illustrates an example process flow according to a possibleembodiment of the present invention. In some possible embodiments, oneor more computing devices or components such as an image processingsystem (e.g., 100) may perform this process flow. In block 412, theimage processing system (100) may generate first measurable imageryresponses of a scene. The first measurable imagery responses may beobtained by exposing a first field of a multi-field image sensor of animage processing system for a first time duration.

In block 414, the image processing system (100) may generate secondmeasurable imagery responses of the scene. The second measurable imageryresponses at least in part may be obtained by exposing a second field ofthe multi-field image sensor of the image processing system for a secondtime duration that contains the first time duration.

In block 416, the image processing system (100) may convert the firstmeasurable imagery responses and the second measurable imagery responsesinto at least a part of an output image of the scene. The output imagemay have a higher dynamic range than either of the first measurableimagery responses and the second measurable imagery responses.

In some possible embodiments, the first field and the second fieldmentioned above may be interleaved.

In some possible embodiments, the second measurable imagery responsesmay further comprise measurable imagery responses in the first field fora time duration equal to a difference between the first time durationand the second time duration.

In some possible embodiments, the image processing system (100) may beconfigured with a color space; the first measurable imagery responsesmay comprise imagery responses for one or more component colors that arefewer than a full complement of component colors in the color space.

In some possible embodiments, the image processing system (100) maygenerate third measurable imagery responses of the scene; the thirdmeasurable imagery responses at least in part may be obtained byexposing a third field of the multi-field image sensor of the imageprocessing system for a third time duration that contains the secondtime duration. In some such embodiments, to convert the first measurableimagery responses and the second measurable imagery responses into atleast a part of an output image of the scene, the image processingsystem (100) may convert the first measurable imagery responses, thesecond measurable imagery responses and the third measurable imageryresponses into the output image.

In some possible embodiments, the image processing system (100) may beconfigured with a color space. In some such embodiments, a fullcomplement of component colors in the color space may comprise componentcolors from the first measurable imagery responses and the secondmeasurable imagery responses.

In some possible embodiments, the image processing system (100) maygenerate, based on (1) the first measurable imagery responses and (2)the second measurable imagery responses, a first image of the scene,wherein the first image may have the full complement of component colorsin the color space. Further, the image processing system (100) maygenerate, based on (1) the first measurable imagery responses, (2) thesecond measurable imagery responses, and (3) the third measurableimagery responses, a second image of the scene, wherein the second imagemay have the full complement of component colors in the color space. Insome such embodiments, to convert the first measurable imageryresponses, the second measurable imagery responses and the thirdmeasurable imagery responses into the output image, the image processingsystem (100) may combine the first image and the second image into theoutput image of the scene.

In some possible embodiments, the image processing system (100) may beconfigured with a color space; a full complement of component colors inthe color space may comprise component colors from the first measurableimagery responses and the second measurable imagery responses.

In some possible embodiments, the image processing system (100) maygenerate, based at least in part on the first measurable imageryresponses, a first image of the scene, wherein the first image may havethe full complement of component colors in the color space. Further, theimage processing system (100) may generate, based on (1) the firstmeasurable imagery responses and (2) the second measurable imageryresponses, a second image of the scene, wherein the second image mayhave the full complement of component colors in the color space. In somesuch embodiments, to convert the first measurable imagery responses andthe second measurable imagery responses into at least a part of anoutput image, the image processing system (100) may combine the firstimage and the second image into the output image of the scene.

In some possible embodiments, one or more pixels values for at least onecomponent color in the first image are predicted. In some possibleembodiments, the one or more pixel values may be predicted with one ormore histograms. In some possible embodiments, a histogram as describedherein may be derived in part from one or more of the first measurableimagery responses and the second measurable imagery responses. In somepossible embodiments, a histogram as described herein may be derivedfrom neither the first measurable imagery responses nor the secondmeasurable imagery responses. In some possible embodiments, a histogramas described herein may be derived in part from an image in an imagecollection. In some possible embodiments, one or more pixel values forat least one component color in the first image may be predicted with ablack-body radiation spectrum.

FIG. 4C illustrates an example process flow according to a possibleembodiment of the present invention. In some possible embodiments, oneor more computing devices or components such as an image processingsystem (e.g., 100) may perform this process flow. In block 422, theimage processing system (100) may receive a first image of a scene,wherein the first image may have a first dynamic range.

In block 424, the image processing system (100) may receive a secondimage of the scene, wherein the second image may have a second differentdynamic range.

In block 426, the image processing system (100) may predict, based atleast in part on the second image, one or more pixel values of the firstimage for a component color.

In block 428, the image processing system (100) may generate, based atleast in part on the first image and the one or more pixel valuespredicted based at least in part on the second image, an output image ofthe scene, wherein the output image may have a higher dynamic range thaneither of the first image and the second image.

In some possible embodiments, the component color mentioned above may bemissing in the first image.

In some possible embodiments, the first image may comprise at least oneof overexposed pixels and under-exposed pixels.

In some possible embodiments, the pixel values for the component colormay be predicted with one or more histograms at least one of which isderived in part from the second image.

In some possible embodiments, the pixel values for the component colormay be predicted with one or more histograms at least one of which isderived in part from an image in an image collection.

In some possible embodiments, at least one of the pixel values for thecomponent color may be predicted with a black-body radiation spectrum.

10. IMPLEMENTATION MECHANISMS Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 5 is a block diagram that illustrates a computersystem 500 upon which an embodiment of the invention may be implemented.Computer system 500 includes a bus 502 or other communication mechanismfor communicating information, and a hardware processor 504 coupled withbus 502 for processing information. Hardware processor 504 may be, forexample, a general purpose microprocessor.

Computer system 500 also includes a main memory 506, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 502for storing information and instructions to be executed by processor504. Main memory 506 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 504. Such instructions, when stored innon-transitory storage media accessible to processor 504, rendercomputer system 500 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 500 further includes a read only memory (ROM) 508 orother static storage device coupled to bus 502 for storing staticinformation and instructions for processor 504. A storage device 510,such as a magnetic disk or optical disk, is provided and coupled to bus502 for storing information and instructions.

Computer system 500 may be coupled via bus 502 to a display 512, such asa liquid crystal display, for displaying information to a computer user.An input device 514, including alphanumeric and other keys, is coupledto bus 502 for communicating information and command selections toprocessor 504. Another type of user input device is cursor control 516,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 504 and forcontrolling cursor movement on display 512. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 500 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 500 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 500 in response to processor 504 executing one or more sequencesof one or more instructions contained in main memory 506. Suchinstructions may be read into main memory 506 from another storagemedium, such as storage device 510. Execution of the sequences ofinstructions contained in main memory 506 causes processor 504 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 510.Volatile media includes dynamic memory, such as main memory 506. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 502. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 504 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 500 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 502. Bus 502 carries the data tomain memory 506, from which processor 504 retrieves and executes theinstructions. The instructions received by main memory 506 mayoptionally be stored on storage device 510 either before or afterexecution by processor 504.

Computer system 500 also includes a communication interface 518 coupledto bus 502. Communication interface 518 provides a two-way datacommunication coupling to a network link 520 that is connected to alocal network 522. For example, communication interface 518 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 518 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 518sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 520 typically provides data communication through one ormore networks to other data devices. For example, network link 520 mayprovide a connection through local network 522 to a host computer 524 orto data equipment operated by an Internet Service Provider (ISP) 526.ISP 526 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 528. Local network 522 and Internet 528 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 520and through communication interface 518, which carry the digital data toand from computer system 500, are example forms of transmission media.

Computer system 500 can send messages and receive data, includingprogram code, through the network(s), network link 520 and communicationinterface 518. In the Internet example, a server 530 might transmit arequested code for an application program through Internet 528, ISP 526,local network 522 and communication interface 518.

The received code may be executed by processor 504 as it is received,and/or stored in storage device 510, or other non-volatile storage forlater execution.

11. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

In the foregoing specification, possible embodiments of the inventionhave been described with reference to numerous specific details that mayvary from implementation to implementation. Thus, the sole and exclusiveindicator of what is the invention, and is intended by the applicants tobe the invention, is the set of claims that issue from this application,in the specific form in which such claims issue, including anysubsequent correction. Any definitions expressly set forth herein forterms contained in such claims shall govern the meaning of such terms asused in the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

For the purpose of illustrations only, it has been described that insome possible embodiments, two images with different exposure may beused to create an HDR image. It should be noted that other numbers ofimages may be used to create an HDR image. In some possible embodiments,a single exposure is sufficient to create an HDR image. For example, amissing primary color for the image with the single exposure may bepredicted or estimated. In some possible embodiments, more than two (2)exposures may be used to create an HDR image. In addition, numbers ofexposures to be used in creating an HDR image may vary with specificpixels. On some pixels, only one exposure may be used, while on someother pixels two or more exposures may be used. When more than twoexposures are used to create pixels for the HDR image, the selection ofpixels from which exposure may depend on the values of the pixels. If apixel in a longer exposure is beyond a range, e.g., covered by thelonger exposure, a pixel from a shorter exposure may be used toestimate, interpolate or predict an appropriate pixel value in the HDRimage. For example, the pixel in the shorter exposure may be scaled upin magnitude by a factor as determined by relative exposure timesassociated with the different exposures.

Under techniques herein, images with short and long exposures may becombined or used to generate an HDR. If a long exposure image is notclipped, then pixels in the long exposure image may be used to generatecorresponding pixels in the HDR without using shorter exposures.

On the other hand, if pixels in the long exposure image are clipped,then pixels in a short exposure image may be used to estimate, predict,or interpolate corresponding pixels in the HDR.

It should be noted that for the purpose of the present invention, theroles described for long and short exposures may be reversed. Forexample, instead of using short exposures to predict pixel values in along exposure, long exposures may be used to predict pixel values in ashort exposure for the purpose of creating an HDR image.

What is claimed is:
 1. A method comprising: using a multi-field imagesensor to generate a first image of a scene with a first operationalmode of an image processing system; using the multi-field image sensorto generate a second image of the scene with a second differentoperational mode of the image processing system, the first image beingof a first spatial resolution, the second image being of a secondspatial resolution, and the first spatial resolution being higher thanthe second spatial resolution; wherein a first time duration in thefirst operational mode used to generate the first image of the sceneoverlaps with a second time duration in the second operational mode usedto generate the second image of the scene; upsampling the second imageto a stack frame that comprises a same number of pixels as the firstimage; combining the first image and the second image into an outputimage of the scene, the output image being of a higher dynamic rangethan either of the first image and the second image and of a spatialresolution higher than the second spatial resolution.
 2. The method ofclaim 1, wherein the first operational mode is a full resolutionoperational mode of the image processing system, and wherein the secondoperational mode is a low resolution mode, in relation to the firstoperational mode, of the image processing system.
 3. The method of claim1, wherein the first image is generated with a first exposure, andwherein the second image is generated with a second exposure.
 4. Themethod of claim 3, wherein the first exposure is time-wise longer thanthe second exposure.
 5. The method of claim 3, wherein the firstexposure is time-wise shorter than the second exposure.
 6. The method ofclaim 1, wherein a first pixel in the output image has a first luminancevalue derived from one or more first luminance values for one or morefirst pixels in the first image, and wherein a second pixel in theoutput image has a second luminance value derived from one or moresecond luminance values for one or more second pixels in the secondimage.
 7. A method comprising: using a multi-field image sensor togenerate a first image of a scene, the first image having a firstdynamic range; using a multi-field image sensor to generate a secondimage of the scene, the second image having a second different dynamicrange; wherein a first time duration in a first operational mode used togenerate the first image of the scene overlaps with a second timeduration in a second different operational mode used to generate thesecond image of the scene; predicting, based at least in part on thesecond image, one or more pixel values of the first image for acomponent color; wherein the component color is missing in the firstimage; generating, based at least in part on the first image and the oneor more pixel values predicted based at least in part on the secondimage, an output image of the scene, the output image having a higherdynamic range than either of the first image and the second image. 8.The method of claim 7, wherein the first image comprises at least one ofoverexposed pixels and under-exposed pixels.
 9. The method of claim 7,wherein the pixel values for the component color are predicted with oneor more histograms at least one of which is derived in part from thesecond image.
 10. The method of claim 7, wherein the pixel values forthe component color are predicted with one or more histograms at leastone of which is derived in part from an image in an image collection.11. The method of claim 7, wherein at least one of the pixel values forthe component color is predicted with a black-body radiation spectrum.12. An apparatus comprising a processor and configured to perform: usinga multi-field image sensor to generate a first image of a scene with afirst operational mode of an image processing system; using themulti-field image sensor to generate a second image of the scene with asecond different operational mode of the image processing system, thefirst image being of a first spatial resolution, the second image beingof a second spatial resolution, and the first spatial resolution beinghigher than the second spatial resolution; wherein a first time durationin the first operational mode used to generate the first image of thescene overlaps with a second time duration in the second operationalmode used to generate the second image of the scene; upsampling thesecond image to a stack frame that comprises a same number of pixels asthe first image; combining the first image and the second image into anoutput image of the scene, the output image being of a higher dynamicrange than either of the first image and the second image and of aspatial resolution higher than the second spatial resolution.
 13. Theapparatus of claim 12, wherein the first operational mode is a fullresolution operational mode of the image processing system, and whereinthe second operational mode is a low resolution mode, in relation to thefirst operational mode, of the image processing system.
 14. Theapparatus of claim 12, wherein the first image is generated with a firstexposure, and wherein the second image is generated with a secondexposure.
 15. The apparatus of claim 14, wherein the first exposure istime-wise longer than the second exposure.
 16. The apparatus of claim12, wherein the first exposure is time-wise shorter than the secondexposure.
 17. The apparatus of claim 12, wherein a first pixel in theoutput image has a first luminance value derived from one or more firstluminance values for one or more first pixels in the first image, andwherein a second pixel in the output image has a second luminance valuederived from one or more second luminance values for one or more secondpixels in the second image.
 18. An apparatus comprising a processor andconfigured to perform: using a multi-field image sensor to generate afirst image of a scene, the first image having a first dynamic range;using the multi-field image sensor to generate a second image of thescene, the second image having a second different dynamic range; whereina first time duration in a first operational mode used to generate thefirst image of the scene overlaps with a second time duration in asecond different operational mode used to generate the second image ofthe scene; predicting, based at least in part on the second image, oneor more pixel values of the first image for a component color; whereinthe component color is missing in the first image; generating, based atleast in part on the first image and the one or more pixel valuespredicted based at least in part on the second image, an output image ofthe scene, the output image having a higher dynamic range than either ofthe first image and the second image.
 19. The apparatus of claim 18,wherein the first image comprises at least one of overexposed pixels andunder-exposed pixels.
 20. The apparatus of claim 18, wherein the pixelvalues for the component color are predicted with one or more histogramsat least one of which is derived in part from the second image.
 21. Theapparatus of claim 18, wherein the pixel values for the component colorare predicted with one or more histograms at least one of which isderived in part from an image in an image collection.
 22. The apparatusof claim 18, wherein at least one of the pixel values for the componentcolor is predicted with a black-body radiation spectrum.